Succinct Sampling on Streams

A streaming model is one where data items arrive over long period of time, either one item at a time or in bursts. Typical tasks include computing various statistics over a sliding window of some fixed time horizon. What makes the streaming model interesting is that as the time progresses, old items expire and new ones arrive. One of the simplest and most central tasks in this model is sampling. That is, the task of maintaining up to k uniformly distributed items from a current time-window. We call sampling algorithms succinct if they use provably optimal (up to constant factors) worst-case memory to maintain k items (either with or without replacement). We stress that in many applications, structures that have expected succinct representation as the time progresses are not sufficient. That is, expected small memory solutions in a streaming environment will never provide a fixed bounded memory guarantee over the lifetime of a (very large) stream, as small probability events eventually happen with probability 1. Thus, in this paper we ask the following question: Are succinct sampling on streams (or S algorithms) possible, and if so, for what models? Perhaps somewhat surprisingly, we show that S algorithms (i.e. with matching upper and lower bounds and worst case fixed memory guarantees) are possible for all variants of the problem mentioned above, i.e., both with and without replacement and both for one-at-a-time and bursty arrival models. In addition to fixed memory guarantees, our solution has additional benefits that are important in applications: in “one-item-at-a-time” model, the samples produced over non-overlapping windows are completely independent of each other (this was not the case for previous solutions) and in the bursty model, previous solutions required floating point computations; we do not. Finally, we use S algorithms to solve various problems in the sliding windows model, including frequency moments, counting triangles, entropy and density estimations. For these problems we present first solutions with provable worst-case memory guarantees. The results that we arrive at are based on the novel sampling method that could be of independent interest.